MEDIA COMPOSITION SYSTEM WITH KEYBOARD-BASED EDITING CONTROLS 
[0001] Cross Reference to Related Applications 

[0002] This application is a continuation of Ser. No. 09/489,330, January 21, 2000, which is a 
continuation of Ser. No. 08/873,577, June 12, 1997, Pat No. 6,018,337, which is a continuation 
of Ser. No. 08/045,646, April 9, 1993, which is a continuation-in-part of Ser. No. 07/866,829, 
Apr. 10, 1992, Pat. No. 5,355,450, all of which are herein incorporated by reference. 

[0003] Field of the Invention 

[0004] The invention relates to the display of sequentially stored samples under the control of a 
pointer and/or keyboard, and more particularly, to the use of a mouse and/or keyboard as a jog or 
shuttle control for computer — based video and audio composition. 

[0005] Background of the Invention 

[0006] It is known to provide on a media player, such as a video tape recorder, a jog and/or 
shuttle knob. A shuttle knob allows the operator of the media player to adjust the speed of the 
player in either a forward or reverse direction. A jog control allows the user to move in forward 
and reverse directions through the medium on a still frame by frame basis. Examples of prior art 
jog and shuttle controls include levers, knobs, concentric knobs, and knobs that change function 
when pressed upon axially or when another switch is pressed. 

[0007] A common system is a three — button system, in which the user places three of his or 
her fingers on three buttons. The outer two fingers control the forward and reverse shuttle 
operations, which are much like fast forward scan and rewind scan buttons on a video tape 
recorder, except that multiple presses on the outer buttons in shuttle mode result in proportional 
increases in shuttling speed. For example, three presses will cause shuttling to take place at triple 
the shuttling speed. The central button is a stop button, and when it is held down, the outer 
buttons act as forward and reverse jog controls. 

[0008] In computer — based media composition systems, a visual representation of a shuttle or 
jog control may be displayed and manipulated with a mouse. For example, an image of a slider 
or a film strip may be presented to the user. The user may then click on the slider with the mouse 
and drag the slider to perform either shuttle or jog operations. For example, the Quantel Harry 
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system, available from Quantel Systems, Inc. of England, uses this type of metaphor. Generally, 
however, these systems require the user to be watching the screen so that they may locate the 
mouse pointer in a proper position on the slider. 

[0009] Summary of the Invention 

[0010] In general, the invention features selecting samples for presentation on an output 
device, such as a display or speaker, from a sequence of stored media samples, such as audio or 
video information. Position information is received from a pointing device, such as a mouse, and 
translated into direction and magnitude information. A second sample is then retrieved based on 
this position and magnitude information. This method may be used to implement jog or shuttle 
controls for a media composition system, which may be provided with simulated "inertia" for 
ease of use. 

[0011] A media composition system according to the invention has the advantage of 
convenience and efficiency for the user. The user may perform composition operations using the 
pointing device and, without removing his or her hand from the pointing device, move to the 
next location on the medium to be composed. The user may also do so without looking at a 
control screen at all; he or she need only look at the material to be composed. These capabilities 
are provided inexpensively in a commonly — available pointing device, which may already be a 
part of the user's composing system. Also, the added inertia in both jog and shuttle modes, and 
playback speed limit in shuttle mode may prevent the user from getting lost in the material to be 
composed. 

[0012] Brief Description of the Drawings 

[0013] For a better understanding of the present invention, together with other and further 
objects, advantages and capabilities thereof, reference is made to the accompanying drawings, 
which are incorporated herein by reference and in which: 

[0014] FIG. 1 presents a media composing workstation portion of a media composing system 
according to the invention; 

[0015] FIG. 2 is a system block diagram of the media composing system of FIG. 1; 

[0016] FIG. 3 is an exemplary flowchart illustrating operation of a workstation portion such as 

the one shown in Figs. 1 and 2; 
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[0017] FIG. 4 is another exemplary flowchart illustrating operation of a workstation portion 
such as the one shown in Figs. 1 and 2; 

[0018] FIG. 5 is a further exemplary flowchart illustrating operation of a workstation portion 
such as the one shown in Figs. 1 and 2; 

[0019] FIG. 6 is a schematic diagram of another media composition system; 
[0020] FIG. 7 is a schematic view of the motion effects screen; 
[0021] FIGS. 8a and 8b are schematic illustrations of the transition effects screen; 
[0022] FIG. 9 is an illustration of a keyboard layout; and 

[0023] FIG. 10 is a perspective view of the mechanical user interface according to the 
invention. 

[0024] Description of the Preferred Embodiment 

[0025] Referring to FIG. 1, an exemplary media composition workstation according to the 
invention includes a computer output system 12 and computer input system 14. The output 
system may include one or more computer monitors 16 and one or more speakers 18. The input 
device may include a keyboard 20 and a pointing device 22, such as a mouse with one or more 
buttons 32. The display presented to the user on the monitor may be divided into a window 24 
for variable speed playback, on — screen scrub controls 26, a timeline display 28, and a timeline 
position indicator 30. 

[0026] Referring to FIG. 2, a media composing system 40 according to the invention may also 
comprise a mouse/scrub control module 42, a player module 44, a timeline module 46, and 
storage 48. The mouse/scrub control module is connected to the keyboard via a data path 50 and 
to the pointing device by another data path 52. The player module 44 is connected to the 
mouse/scrub control module via a further data path 54. 

[0027] The player module is also connected to receive information from the storage 48 via a 
bus 56, which may comprise several data lines 64, 66, 68. These data lines may be dedicated to 
different synchronized media channels, such as video and audio channels, The bus 56 is further 
routed to the timeline module 46, which also receives data from the player module via a data 
path 58. The player module may provide information to the moving picture window 24 via a 
picture information path 60, and may provide synchronized sound information to the speaker via 
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a sound information data path 62. The timeline module supplies information to the timeline 
display 28 via a timeline information path 70. 

[0028] It is observed that the routing of data and information may differ from the block 
diagram in an actual implementation. For example, the moving picture window and timeline 
display may be displayed on the same monitor and therefore share a data line. The pointer data 
might also be routed through the keyboard on its way to the mouse/scrub module or the keyboard 
might not be needed at all. Furthermore, the invention herein is applicable to composition 
operations on sound alone, and therefore the displays may not be necessary. Alternatively, the 
operations may be performed on images alone, making speakers unnecessary. 
[0029] The pointing device 22 may be one of several general purpose pointing devices 
available for computers. It may be a mouse supporting one or more buttons 32, or it may be a 
trackball, a light pen, a digitizing pad, touch pad, touch screen or the like. These types of 
pointing devices are commonly used to position a cursor on a screen of a computer system, or to 
receive position dependent data, such as that generated in a manual digitizing operation. 
[0030] A system according to the invention may be developed using a variety of types of 
computer systems. In one embodiment, an Apple Macintosh and the "C" computer language 
were used, but naturally the invention should not be read to be limited to this particular platform. 
The mouse/scrub control module, player module, and timeline module may be implemented in 
software and/or hardware and may or may not be contained within the computer system. A 
software package entitled "Avid Media Composer ®", available from Avid Technology, Inc. of 
Tewksbury, Massachusetts includes these features. The storage may include a high-speed, high 
capacity internal magnetic hard disk drive. For further information on computer — based media 
storage and playback of video and audio information, see Peters et al., "VIDEO AND AUDIO 
TRANSMISSION SYSTEM AND METHOD", U.S. patent No. 5,045,940, issued September 3, 
1991, herein incorporated by reference. 

[0031] Storage 48 contains one or more sequences of image and/or sound samples, and may 
also contain a sequence of composition information, such as editing and special effect 
information. The images may be stored in compressed form, allowing more images to be stored 
in a given amount of space. The sequences of images and sound are generally related in time 
much like the tracks of a multiple track type recording. For example, the storage may provide 
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multiple logical connections including one or more for picture information 64, one or more for 
sound information 66, and one or more for composition information 68. 
[0032] Referring to FIGS. 1 and 2, in composing material, the user interacts with the 
workstation portion of the system 10. Composition operations may include editing operations, 
special effects generation, dubbing operations, or the like. The composition to be worked on is 
displayed in an outline format in the timeline display. This outline format may include 
waveforms for sound information, or a series of representative pictures for image information 
such as video or film sequences. If a user desires to perform a composing operation on a 
particular scene he, or she may move the pointer 34 on the screen using the mouse and click on 
the portion of the timeline which shows the representative scene indicating the location at which 
the edit is to be performed. This is a relatively rough operation as computer displays will 
generally only allow display of a relatively low number of images on a timeline at a meaningful 
size. Once the user has located a rough location for the edit using the timeline position indicator, 
he or she may perform more precise jog and shuttle operations. To do so, the operator may use 
the pointing device to click on the on — screen controls 26 or may press a key 21 on the keyboard 
20. This operation puts the workstation in shuttle mode. In this mode, when the user moves the 
mouse to the left, the workstation begins moving through the stored sequence of images at a rate 
determined by the distance through which the user has moved the mouse. This corresponds to a 
reverse shuttle operation, which may analogized to scan rewinding a video tape at a variable rate, 
with the rate dependent on the mouse displacement. Conversely, moving the mouse to the right 
will cause the sequence of images to be moved through in a forward direction in a forward 
shuttle operation. 

[0033] The system 40 performs this shuttle operation by first obtaining position information 
from the pointing device through the data path 52. This operation may be performed by directly 
accessing the hardware associated with the pointing device, or by retrieving a coordinate stored 
by the computer's operating system. This mouse position information is then translated by the 
mouse/scrub control module into direction and magnitude information, which is transferred to 
the player module. 

[0034] The magnitude information may be represented by the X — coordinate of the current 
position of the mouse subtracted from the X — coordinate of the position of the mouse when the 
mode was entered, with this difference being multiplied by a scaling constant. The sign of this 
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difference may be used as the direction information. The value of the constant will determine 
how sensitive the shuttle operation will be. 

[0035] The direction and magnitude information can be transformed in this operation into 
numerical values / which can be directly applied to the player module as commands. The 
direction information represents the direction of playback for the player module, and the 
magnitude information represents the speed of playback. These relayed commands directly cause 
the player module to display the sequences of images and/or sounds on the display at the 
appropriate rate. 

[0036] In the shuttle mode, the system limits the maximum forward or reverse speed to three- 
times the sound speed (the sound speed is the speed at which a video sequence is played where 
the sound is played at the proper speed). This speed limiting feature prevents the user from 
getting lost by moving the mouse at a high rate of speed, such as by accidentally knocking the 
mouse across the desk. 

[0037] Furthermore, in shuttle mode, "inertia" is added to the operation of the control. Rather 
than supplying information about mouse position changes directly to the player, changes in 
mouse position are moderated over a short period. In one embodiment this is done by responding 
to changes in mouse position by providing a smaller corresponding change in speed to the player 
module and gradually increasing that change in speed to the full change in speed indicated by the 
mouse. For example, if the user were to double the distance of the mouse from its starting 
position, the mouse scrub control module might convey a 50% increase to the player module, 
followed closely by a 75% increase, and then a 90% increase, and finally a 100% increase. It has 
been found that this inertia enhances the ease of use and efficiency of use of the apparatus. 
[0038] In shuttle mode, the system also has a small threshold value below which no shuttle 
operation takes place. This prevents the player from playing back at extremely slow speeds 
which are of little use and are sometimes found to be uncomfortable to the user. Values on the 
order of a 1% threshold have been found to be satisfactory. 

[0039] The user may also use a jog mode for the workstation. Jog mode is similarly controlled 
by the position of the mouse, but advances from still frame to still frame within the sequence of 
images, rather than adjusting the speed of playback of the images. In jog mode, the distance the 
mouse travels from its starting point relates to the offset within the sequence of frames between 
the current position and the position to be displayed. 
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[0040] To enter jog mode from shuttle mode, the user may simply press and hold the mouse 
button 32. This stops the action in the display window and subsequent motion of the mouse will 
constitute jogging operations. Other methods for entering the jogging mode are possible, such as 
keyboard keys 21. Similarly, the user may exit the jog mode by one of these methods, or by 
double clicking on the mouse button 32. Releasing the mouse button while in jog mode will 
return the user to shuttle mode, and with the current mouse position as the starting position at 
zero play speed. 

[0041] In the jog mode, the mouse scrub control module receives position information from the 
pointing device and translates it into a different kind of direction and magnitude information. 
The mouse scrub control module constantly monitors the mouse and performs velocity 
calculations based on the position information of the mouse over time. The resulting velocity 
information may be supplied directly to the player module, after it is adjusted by a constant. By 
making the speed of the player proportional to that of the mouse, the player module will track the 
mouse position. 

[0042] Inertia may also be added to the jog operations in a manner similar to that disclosed 
above in connection with the shuttle mode. This is particularly useful in connection with sound 
information, as it makes sound playback more regular during a jog operation. 
[0043] It is noted that the timeline moves during both jog and shuttle operation and that this 
provides feedback to the user in these operations. This feedback enhances the ease of use of the 
workstation operations and prevents the user from getting lost. Also, the user or system designer 
may adjust the constants determining the sensitivity of the jog and shuttle controls and the 
inertia, threshold and maximum speeds in the shuttle mode. This allows the user or designer to 
customize the "feel" of the workstation. 

[0044] With reference to FIG. 6 another media composition system 1 10 includes a pair of CRT 
displays 1 12 and 1 14 and an audio output device 1 16 which may include two speakers. The 
video displays 1 12 and 1 14 and the audio transducer 1 16 are controlled by a computer 1 18. It is 
preferred that the computer 1 18 be a Macintosh from Apple Computer Corporation designated as 
II X , Hci, Hfx or Quadra 900. Disk storage apparatus 120 communicates with the computer 118. 
Disk storage 120 includes one to seven disks for media storage. The disks may be optical or 
magnetic. The system 1 10 is controlled by a keyboard 122 and a mechanical user interface 124 
to be described in more detail herein. 
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[0045] In operation, video and audio source material is received by the system 1 10, digitized 
and stored in the disk storage device 120. The computer 1 18 is programmed so that the digitized 
source material may be edited and displayed on one of the video display devices such as the CRT 
display 1 12. Typically digitized source material would be displayed at a location 126 and edited 
material at a location 128 on the display 112. 

[0046] The improved media composition system of the invention allows the user to pre- 
visualize motion effects by creating clips and media files which display the requested effect. The 
new clip can be used like any other chip—it can be trimmed, extracted, overwritten, used in 
dissolves and wipes, etc. -and its media can be manipulated as any other, that is, it can be 
deleted, consolidated, and even back digitized. The new motion effect clips are video only. The 
start time code is zero hour regardless of the time code of the original clip. Motion effects can be 
created from master clips and subclips, but not from other motion effect clips. There is a delay as 
the media composition system creates new media files. Motion effects are organized into three 
related features: variable speed, fit-to-fill, and strobe motion. These features are accessed 
through a single command in a source menu. A Motion Effects command opens a dialog box 
illustrated in FIG. 7. A preview dial 134 allows the operator to pre-visualize the effect even 
before the OK function 136 is clicked on. The dial 134 normally rotates at one revolution per 
second. When the operator enters values for an effect and clicks on PREVIEW, the dial rotates at 
the new appropriate speed. In this way, the preview dial works as a metronome to give the 
operator a feel for the pace or rhythm of the effect. 

[0047] Forward and reverse variable speed effects will now be discussed. First of all, the 
operator opens a clip in the source monitor 1 12 and marks an IN and an OUT, and chooses 
motion effects from the monitor 1 14 command menu. The operator then enters any one of three 
parameters: duration, play rate (in fps), or percent speed. When any one of these parameters is 
entered, the media composition system immediately calculates and displays the values for the 
other two parameters. For example, if one marks a one second piece and enter 50% fps, the 
media composition system will immediately show two seconds as the duration and 15 fps as the 
play rate of the new clip. One specifies reverse motion by entering an negative rate or percent 
speed. When the operator clicks on OK, the media composition system creates a new clip and 
video/media file for the appropriate effect and loads this clip into the source monitor 1 12. It 
should be noted that when the operator asks for slow motion, the media composition system 
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creates a clip in which each frame is duplicated a proportionate number of times. Similarly, when 
one asks the media composition system for fast motion, it creates a clip in which some portion of 
the source frames have been deleted. When these clips are played, the motion may appear jerky 
since the media composition system is not synthesizing a smoothed series of frames. This effect 
is especially likely with material transferred from film. 

[0048] The fit-to-fill option allows one to have the media composition system calculate the 
motion effect required-to fill a specific duration in a sequence with a specific duration from the 
source. The fit-to-fill check box 138 is bold only when the operator has marked an IN and OUT 
in both monitors 126 and 128 or the four marks are implied by the location of the position 
control. Given these values for the source (current) and target (new) durations, the media 
composition system calculates the necessary rate in percent speed of the motion effect. One may 
accept these or enter one's own values. Once the media composition system has created a new 
clip, one can overwrite or splice it into the sequence. 

[0049] Strobe motion is a type of motion effect in which one frame is held for a specific 
duration and then the next for the same duration and so on. When one opens a clip in the source 
monitor, mark an IN and an OUT, and select strobe motion 140 in the motion effects dialog box. 
The operator must fill in the n in "update every n frames" 142. The new clip will have the same 
duration as the current one but only every nth frame is displayed. A Remake Effects command 
recreates dissolve media files and includes all motion effects. As with transition effects it is only 
possible to recreate motion effect media files when the original source media is online. When the 
operator batch digitizes a sequence which includes motion effects, the effects are automatically 
recreated at the end of the process. Motion effects are represented correctly in all supported edit 
decision list (EDL) formats with the following caveat: It may not be possible to express 
accurately the media composition system motion effect in an EDL because the format may limit 
the precision with which percent speed can be expressed to whole numbers or one decimal place 
and the media composition system is not likewise limited. In this case, the media composition 
system appropriately truncates the rate or percent speed in the EDL and generates a comment 
which indicates the true rate or percent speed. With the exception of freeze flames, motion 
effects are ignored by auto-assembly. Auto-assembly edits video from an original clip at 30 
frames per second, starting at the edit inpoint and filling the duration of the effect. 
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[0050] The media composition system of the invention offers wipes as a transition effect. 
Wipes are accessed through a Transition Effect command. This command opens a dialog box 
144 (FIG. 8a) which allows the user to choose in a pop-up menu between the two transition 
effects: dissolve and wipe. When wipe is selected, the operator can choose a pattern from a menu 
of sixteen choices 146 (displayed graphically) and a direction-forward or reverse as shown in 
FIG. 8b. Forward means that the outgoing clip is represented by the white in the pattern from the 
menu 146 and the incoming by the black (actually blue). Reverse means the incoming is 
represented by the white. Regardless of whether the operator chooses dissolve or wipe, the 
duration must be entered in frames, its start relative to the transition (starting, centered, ending, 
or offset) and a target drive for the media files. 

[0051] The effect can be viewed only after the media composition system creates a media file 
for the specified wipe or dissolve. These effect files will be created, deleted, and recreated in 
exactly the same way dissolve media files have been in earlier versions. The Remake Effects 
command includes all transition effects. As with motion effects, it is only possible to recreate 
transition effect media files when the original source media is online. For example, media for 
both the incoming and outgoing clip must be online for the media composition system to recreate 
the dissolve between them. When a sequence which includes transition effects is digitized, the 
effects are automatically recreated at the end of the process. All wipes are expressed correctly in 
all EDL formats. A dialog box from the EDL Tool allows one to specify the appropriate pattern 
number for each wipe pattern. The table of numbers and patterns is stored in a file which can be 
moved from one media composition system to another. It is not, however, possible to save and 
choose among several different sets of values. The present media composition system will also 
allow the operator to zoom to full-screen mode from any monitor (source, record, pop-up) by 
pressing the quote key. All keyboard equivalents function in full-screen mode. The one 
exception is that one cannot use Trim Mode while in full-screen play. 

[0052] The media composition system of the present invention allows the operator to enter a 
mode in which a mouse controller can be used as a shuttle control. Hit L to shuttle forward, Play 
(the 5 and Back Quote Keys) to shuttle at 30 fps, K (or click a mouse button) to pause (zero 
speed), J to shuttle backward, and the Space Bar to exit the Shuttle Mode. Hit L twice to shuttle 
at 60 fps, thrice to shuttle at 90 fps. Hit J twice for -60 and thrice for -90. While shuttling at zero 
speed, either full-screen or normal, many of the keyboard functions are active. It is possible to 
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step through the program, clear marks, use both kinds of audio scrub (see below), go to the next 
or previous edit, show safe titles, etc. If the media composition system cannot do the function 
and remain in Shuttle Mode, the mode is dropped and the function performed. Toggling between 
source and record is an example of such an operation. In one embodiment of the present 
invention, jog shuttling may be performed, with a result which is similar to that obtained using 
mechanical jog shuttle controls in connection with video tape recorders. With digitized images 
being played, a mouse or similar input device can provide control for jog shuttling. Jog mode 
begins when an operator pressed a button. While the button is depressed, movement of the 
mouse in one direction or another determines the speed of shuttling, or of playback. That is, the 
position of the mouse when the button is depressed is used as a reference position. With a 
relationship defined between position and playing speed, the further the operator moves the 
mouse from the reference position, the faster video is played back in a given direction. For 
example, movement of the mouse to the right increases the forward playing speed. If the mouse 
is returned to the reference position, playing stops. As the mouse is moved to the left, the reverse 
playing speed increases. 

[0053] An important aspect of the present media composition system is a pitch change audio 
scrub feature. When the operator shuttles through footage, smooth, continuous audio will be 
produced at the corresponding speed. That is, pitch will vary with the speed as with an analog 
tape. This feature is available for one track only. Designate the track for smooth scrub by option 
clicking (or double clicking) on the speaker icon for that track. The icon becomes an outline. 
Smooth scrub is available whenever the operator is shuttling, using either the mouse or the 
shuttle control (beneath the Play button on a keyboard discussed below) to determine the shuttle 
speed. 

[0054] Keyboard layout is shown in FIG. 9 and the function of the keys is set forth in Table 1. 
The keyboard 122 is augmented by the mechanical user interface 124, The mechanical user 
interface 124 is shown in more detail in FIG. 10. The interface 124 includes a track ball 150 and 
a speed controller 152. As with a mouse, the track ball 150 may be used to locate a cursor on the 
monitors. Buttons 154 and 156 serve functions like those provided by the click button on a 
mouse. It is preferred that the speed control 152 have detents for zero speed normal forward 
speed (30 frames per second) and a reverse normal speed. It is also preferred that the speed 
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control 152 be spring loaded to return to the zero speed position. It is contemplated that 
additional buttons 158 be provided so that a user may program their functionality. 
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[0055] Yet another aspect of the improved media composition system is sync point editing 
which allows the operator to specify a position in one monitor that is desired to be in sync with 
the position in the other monitor. The operator then performs an overwrite that preserves that 
sync relationship. Sync point editing (SPE) is accessed using the SPE command in the media 
composition system menu. Sync point editing is performed in three steps. First, a point is 
specified in the destination, or record, clip, for example by placing a mouse cursor on the 
displayed frame of the clip and pressing the mouse button. The location within the clip is then 
stored. Second, a point in the source material is identified in a similar manner. Third, the size of 
the source information is specified, for example, by using IN and OUT markers. These steps 
may, in fact, be performed in any order, which may be defined by the programmer of the 
computer. After these three steps are completed, the source information is overwritten in the 
destination, or record, information, by placing the identified source location at the specified 
destination location. Such sync point editing may be performed with any combination of audio 
and video clips. Typically, it is performed to synchronize recorded sound to an event in a video 
clip. The operator may then turn PHANTOM marks on in the media composition system window 
to see how the PHANTOM marks behave in relation to the position control in the record monitor 
128. With SPE off, the system uses the current position as the IN; with SPE on, the current 
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position is the sync point. It should be noted that one can mark the IN and OUT in the record 
monitor 128 instead of the source monitor 126. 

[0056] Another aspect of the invention is slip-sync editing. This kind of editing typically refers 
to maintaining synchronization between a series of video clips and corresponding audio clips 
when transitions between clips are trimmed. In prior art systems, when an audio clip was 
trimmed, i.e., made shorter, subsequent clips became out of synchronization with their 
corresponding video clips. In the present system, when audio material is removed from one end 
of an audio segment from a clip, source material from the original audio clip is added to the other 
end of the segment so as to maintain the length of the audio segment. The source material can 
readily be retrieved from the memory location or disk on which it is stored. Thus, the 
synchronization of subsequent clips is maintained. 

[0057] While there have been shown and described what are at present considered the 
preferred embodiments of the present invention, it will be obvious to those skilled in the art that 
various changes and modifications may be made therein without departing from the scope of the 
invention as defined by the appended claims. 
[0058] What is claimed is: 
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